Introducing new public API fal.realtime.connect

In order to support the new realtime protocol built on top of WebSockets, this PR introduces a new client API. It allows developers to easily interact with model APIs that support realtime inference with very little effort, and also compatible with different JS runtimes and frameworks.

API signature + ergonomics

The API signature is as follows:

function connect(app: string, handler: RealtimeConnectionHandler): RealtimeConnection

It is used as follows:

const connection = fal.realtime.connect('my-superfast-lcm', {
  onResult: (result) => {
    // handle the result
  }
});

connection.send({
  prompt: 'a moon in a starry night sky'
});

Implementation notes

• The client abstracts the usage of WebSocket, the design aims to fit the "duplex realtime communication between client and server", the usage of WS is considered an implementation details the consumer won't need to worry about.
• It handles authentication automatically. WebSockets clients are created in the browser, which make them unsafe when credentials like API keys are needed. Therefore, the client will automatically handle short-lived tokens needed to interact with the fal realtime endpoints (similar to how file uploads are handled).
• It also aims to provide a good/smooth experience out-of-the-box, so it handles particularities of "too many concurrent events", "client-side + server-side rendering", "re-render lifecycles", etc

Event throttling

One of the popular use-cases for LCM/realtime is to generate images as the user draws on a canvas and/or modifies a prompt. However, these events are triggered dozens, sometimes hundreds of times every second. So the client provides a built-in outgoing message throttling mechanism, that makes the experience smooth by default, without overloading the endpoint with too many requests.

const { send } = fal.realtime.connect('my-superfast-lcm', {
  throttleInterval: 128, // default is 64ms
  onResult: handleResult
});

Client+Server rendering

A popular architecture for front-end frameworks nowadays allows developers to share the same code when rendering pages on a server or on a client (e.g. Next, Remix, etc). This enables devs to create rich pages that come populate from the server but are also super interactive on the client. However, this comes with challenges, as developers need to be mindful of some components that are only suitable for client-side rendering or vice-versa. The client is mindful of such use-cases and provides a clientOnly: boolean option that when true will ignore (i.e. no-op) when called on the server-side.

Re-rendering cycles

Another common challenge in popular front-end architectures is to create components that can react to state whenever the frameworks decide to reconcile the data state and the UI representation of such state. This means that a component, that is simply a function, can be called multiple times during its lifecycle, and code that can trigger heavy-weight, long-running operations or manage resources need to be aware of that (e.g. WebSocket connections).

The client introduces a connectionKey option, that when passed makes the client reuse a WebSocket connection available for the same key. This allows devs to simply not worry about rendering cycles, as connect can be called multiple times during the rendering phase that no new connections/listeners/etc will be created.

Here's what a full example of the client working on a Next.js page that does both ssr and csr:

  const { send } = fal.realtime.connect('110602490-lcm-plexed-sd15-i2i', {
    clientOnly: true, // in ssr+csr mode, only run in csr
    connectionKey: 'single-drawing', // reuse connection between render cycles
    throttleInterval: 128, // default is 64, 128 is still smooth but sends less requests
    onResult(result) {
      if (result.images && result.images[0]) {
        setImage(result.images[0].url);
      }
    },
    onError(error) {
      // handle error appropriately
      console.error(error);
    }
  });

Demo

The demo-nextjs-app-router now includes a "realtime" demo page that demonstrates the client usage:

Pending

• Fix create token call
• Proper error handling